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1.0  INTRODUCTION 


With  support  from  the  Environmental  Security  Technology  Certification  Program  (ESTCP), 
Geometries,  Inc.  (Geometries)  is  commercializing  an  advanced  electromagnetic  induction  (EMI) 
system  for  UXO  detection  and  characterization.  Called  the  MetalMapper,  the  new  system  draws 
elements  of  its  design  from  advanced  systems  currently  being  developed  by  G&G  Sciences,  Inc. 
(supported  by  Naval  Sea  Systems  Command,  the  Strategic  Environmental  Research  and 
Development  Program  [SERDP],  and  ESTCP)  and  by  the  Lawrence  Berkeley  National 
Laboratory  (LBNL)  with  support  from  SERDP  and  ESTCP. 

CH2M  HILL  conducted  a  cued  (i.e.  static)  MetalMapper  classification  survey  between  June  20 
and  29,  2011,  as  part  of  the  ESTCP  Live  Site  Demonstrations  at  the  former  Camp  Beale  in 
Marysville,  California.  MetalMapper  data  were  collected  over  1,470  geophysical  targets 
previously  identified  by  EM61-MK2  data  during  an  earlier  phase  of  the  Live  Site 
Demonstrations. 

The  MetalMapper  survey  was  conducted  for  ESTCP  under  contract  W912HQ-1  l-C-0044, 
ESTCP  Project  Number  MR-1063. 

1.1  SURVEY  OBJECTIVES 

The  overall  objective  of  the  MetalMapper  demonstration  was  to  accurately  classify  each  of  the 
targets  as  one  of  the  following: 

•  Target  of  Interest  (TOI),  likely  representing  UXO; 

•  Non-TOI,  likely  representing  another  type  of  anomaly  source  such  as  UXO  fragments,  or 
piece  of  metallic  clutter  unrelated  to  UXO. 

Additional  objectives  for  this  classification  survey  included  the  following: 

•  Selecting  a  dig/no-dig  threshold  that  recognizes  all  TOI  (no  false  negatives/Type  II  error) 
while  minimizing  the  number  of  false  alarms  (minimizing  false  positives/Type  I  error), 

•  Minimizing  the  number  of  targets  classified  as  “can’t  analyze”,  and 

•  Estimating,  correctly,  target  parameters  such  as  polarizabilities,  location,  depth,  and  size. 

2.0  TECHNOLOGY 

2.1  TECHNOLOGY  DESCRIPTION 

A  block  diagram  of  the  data  acquisition  system  (DAQ)  system  is  shown  in  Figure  2-1.  The 
MetalMapper  system  components  include  the  following: 
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1.  Multiple  Transmitter  Loops1 2:  The  MetalMapper  antenna  platform  includes  three 
mutually  orthogonal  transmitter  loops. 

2.  3-Axis  Sensor  Array":  The  MetalMapper  antenna  platfonn  includes  a  spatial  array  of 
seven  3 -axis  receiver  antennas  (resulting  in  21  independent  measurements  of  the  transient 
secondary  magnetic  field). 


A)  MM  DAQ  assembly  with  Panasonic  terminal. 


B)  Function  block  diagram  of  MM  system 


Figure  2-1:  The  MetalMapper  Data  Acquisition  System  and  a  functional  block  diagram 
showing  the  major  components  of  the  overall  system 

3.  Electronically  Switched  Time  Domain  Electromagnetic  Transmitter  Loop  Driver: 

The  transmitter  can  be  controlled  by  the  DAQ  computer  so  that  output  can  be  directed  to 
any  single  loop  or  automatically  multiplexed  between  loops.  Also,  the  fundamental 
waveform  period,  duty-cycle,  and  pulse  polarity  can  be  controlled.  Typically,  however, 
the  loops  are  driven  with  a  classical  bipolar  pulse-type  time-domain  electromagnetic 
(TEM)  waveform  (i.e.,  alternating  pulse  polarity  with  a  50%  duty-cycle).  Depending  on 
the  survey  mode  (e.g.,  static/dynamic),  the  fundamental  frequency  of  transmission  can  be 
varied  over  the  range  1 . 1  l<f  <8 1 0  hertz  (Hz). 


1  The  3  transmitter  loops  is  a  feature  the  MetalMapper  has  in  common  with  the  AOL2,  LBNL  BUD,  and  the  USGS 
ALLTEM  systems. 

2  The  MM  utilizes  3-axis  receiver  antennas,  developed  by  G&G  Sciences  for  the  Navy’s  AOL  system,  to  acquire 
measurements  of  the  vector  magnetic  field.  The  antennas  are  small  and  approximate  an  observation  of  the  field  at  a 
point. 
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The  antenna  array  is  illustrated  in  the  graphic  on  the  right  of  Figure  2-2.  It  has  three  transmitter 
loops  in  the  Z,  Y,  and  X  directions  and  contains  seven  tri-axial  sensors  located  inside  the  Z 
(bottom)  loop.  At  Camp  Beale,  the  antenna  array  was  towed  behind  a  Kubota  B7510  tractor. 
The  antenna  platform  was  positioned  atop  a  sled  that  was  dragged  along  the  ground  surface. 
Using  the  sled,  the  height  of  the  antenna  platfonn  above  the  ground  surface  was  0.16  meters.  It 
is  possible  to  add  wheels  to  the  sled  (not  shown  in  Figure  2-2),  although  for  the  survey  at  Camp 
Beale,  the  sled  was  kept  on  the  ground  surface  in  order  to  minimize  the  height  of  the  antenna 
platform  above  the  geophysical  targets. 

The  DAQ  is  built  around  a  commercially  available  product  from  National  Instruments.  The 
DAQ,  EM  transmitter,  and  batteries  for  the  entire  system  are  packaged  in  an  aluminum  case  that 
was  secured  atop  the  boom  assembly  during  the  survey  at  Camp  Beale. 

In  addition  to  the  components  shown  in  Figure  2-1A,  the  instrumentation  package  includes  a 
real-time  kinematic  global  positioning  system  (RTK-GPS)  receiver  for  recording  positional  data 
and  inertial  measurement  unit  (IMU)  that  records  platfonn  attitude  (i.e.  magnetic  heading,  pitch, 
and  roll).  These  external  devices  are  connected  to  the  DAQ  through  serial  RS232C  ports.  For 
the  survey  at  Camp  Beale,  the  RTK-GPS  base  station  consisted  of  a  Trimble  R8  receiver  and 
Trimble  HPB450  external  radio.  CH2M  HILL  also  utilized  a  Trimble  R8  rover  receiver  with  the 
MetalMapper  system. 

Additional  information  on  the  MetalMapper  specifications,  operational  theory,  and  previous 
classification  studies  conducted  under  the  direction  of  ESTCP  can  be  found  in  the  ESTCP  Live 
Site  Demonstrations  Work  Plan,  MetalMapper  at  Former  Camp  Beale  (CH2M  HILL,  2011). 
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3.0  PERFORMANCE  OBJECTIVES 


The  performance  objectives  for  the  MetalMapper  classification  survey  and  data  analysis  at  Camp 
Beale  site  are  summarized  in  Table  3-1. 

Table  3-1:  Performance  Objectives 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Maximize  correct 
classification  of 
targets  of  interest 
(TOI) 

Number  of  TOI  retained 

•  Prioritized  anomaly 
lists 

•  Ground  truth/scoring 
provided  by  ESTCP/ 
IDA 

Approach  correctly 
classifies  all  TOI 

Maximize  correct 
classification  of 
non-TOI 

Number  of  false  alarms 
eliminated 

•  Prioritized  anomaly 
lists 

•  Ground  truth/scoring 
provided  by  ESTCP/ 
IDA 

Reduction  of  false 
alarms  by  >  50%  while 
retaining  all  TOI 

Specification  of 
no-dig  threshold 

Probability  of  correct 
classification  and 
number  of  false  alarms 
at  demonstrator 
operating  point. 

•  Demonstrator  - 
specified  threshold 

•  Ground  truth/scoring 
provided  by  ESTCP/ 
IDA 

Reduction  of  false 
alarms  by  >  50%  while 
retaining  all  TOI 

Minimize  number 
of  anomalies  that 
cannot  be 
analyzed 

Number  of  anomalies 
that  must  be  classified 
as  “Unable  to  Analyze” 

Demonstrator  target 
parameters 

Reliable  target 
parameters  can  be 
estimated  for  >  98%  of 
anomalies  on  detection 
list 

Correct  estimation 
of  target 
parameters 

Accuracy  of  estimated 
target  parameters 

•  Demonstrator  target 
parameters 

•  Results  of  intrusive 
investigation 

X,  Y  <15  cm  (la) 

Z  <10  cm  (la) 

3.1  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOI 


One  of  the  two  main  objectives  of  this  demonstration  was  to  correctly  classify  all  seeded  items 
and  any  munitions  and  explosives  of  concern  (MEC)  items  remaining  at  the  site  as  TOI  (i.e., 
identify  true  positives,  with  no  false  negatives). 

3.1.1  Metric 

The  metric  for  this  objective  was  the  number  of  items  on  the  MetalMapper  anomaly  list  that  were 
correctly  classified  as  TOI. 
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3.1.2  Data  Requirements 

The  requirement  was  that  MetalMapper  data  were  to  be  analyzed  to  create  a  prioritized  dig  list, 
which  assigned  each  target  to  one  of  four  categories:  0)  Can't  Extract  Reliable  Parameters;  1) 
Likely  TOI;  2)  Can’t  Decide;  or  3)  Likely  Clutter. 

The  targets  classified  as  either  “Likely  TOI”  or  “Can't  Extract  Reliable  Parameters”  were 
considered  “Dig”  targets.  An  additional  number  of  “Can’t  Decide”  targets  down  to  a  “Stop  Dig 
Point”  on  the  prioritized  list  were  selected  to  be  “Dig”  targets  and  those  below  that  point  were 
considered  “Don’t  Dig”.  Institute  for  Defense  Analyses  (IDA)  personnel  used  their  scoring 
algorithms  to  assess  the  results. 

3.1.3  Success  Criteria 

The  objective  was  to  correctly  classify  all  TOI  on  the  prioritized  anomaly  list. 

3.2  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-TOI 

The  second  primary  objective  was  to  discriminate  non-TOI  from  TOI  and  thereby  maximize 
correct  classification  of  non-TOI  that  could  be  eliminated  from  consideration  during  the  intrusive 
investigation  (i.e.,  identify  true  negatives,  with  minimal  false  positives.) 

3.2.1  Metric 

The  metric  for  this  objective  was  the  number  of  targets  on  the  ranked  anomaly  list  correctly 
classified  as  non-TOI. 

3.2.2  Data  Requirements 

The  requirement  was  that  MetalMapper  data  were  to  be  analyzed  to  create  a  prioritized  dig  list, 
which  assigned  each  target  to  one  of  four  categories:  0)  Can't  Extract  Reliable  Parameters;  1) 
Likely  TOI;  2)  Can’t  Decide;  or  3)  Likely  Clutter. 

The  targets  classified  as  either  “Likely  TOI”  or  “Can't  Extract  Reliable  Parameters”  were 
considered  “Dig”  targets.  An  additional  number  of  “Can’t  Decide”  targets  down  to  a  “Stop  Dig 
Point”  on  the  prioritized  list  were  selected  to  be  “Dig”  targets  and  those  below  that  point  were 
considered  “Don’t  Dig”.  IDA  personnel  used  their  scoring  algorithms  to  assess  the  results. 

3.2.3  Success  Criteria 

The  objective  was  to  correctly  label  more  than  50%  of  the  non-TOI  items  as  non-TOI  while 
retaining  all  of  the  TOI  above  the  dig  threshold. 
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3.3  OBJECTIVE:  SPECIFICATION  OF  NO-DIG  THRESHOLD 


For  the  survey  at  the  former  Camp  Beale,  it  is  possible  to  tell  the  true  capabilities  of  a 
classification  procedure  based  solely  on  the  final  ranked  anomaly  list.  However,  all  targets  may 
not  be  dug  for  future  investigations,  so  the  success  of  the  approach  will  depend  on  the  ability  of 
an  analyst  to  accurately  specify  the  dig/no-dig  threshold. 

3.3.1  Metric 

•5 

The  probability  of  correct  classification,  Pciass,  and  number  of  false  alarms,  Nfa,  at  the  dig/no  dig 
threshold  in  the  prioritized  dig  list  were  the  metrics  for  this  objective. 

3.3.2  Data  Requirements 

The  requirement  was  that  MetalMapper  data  were  to  be  analyzed  to  create  a  prioritized  dig  list, 
which  assigned  each  target  to  one  of  four  categories:  0)  Can't  Extract  Reliable  Parameters,  1) 
Likely  TOI,  2)  Can’t  Decide,  or  3)  Likely  Clutter. 

The  category  into  which  each  target  was  placed  was  detennined  using  a  decision  statistic 
developed  during  analysis  of  the  MetalMapper  data.  The  dig/no  dig  threshold  for  this  project 
was  the  decision  statistic  value  that  separated  targets  classified  as  TOI  from  those  classified  as 
non-TOI  (i.e.,  a  value  lying  in  the  “Can’t  Decide”  range).  IDA  personnel  used  their  scoring 
algorithms  to  assess  the  results. 

3.3.3  Success  Criteria 

The  objective  was  to  correctly  label  more  than  50%  of  the  non-TOI  items  as  non-TOI  while 
retaining  all  of  the  TOI  at  the  specified  threshold. 

3.4  OBJECTIVE:  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE 
ANALYZED 

Anomalies  for  which  reliable  parameters  cannot  be  estimated  using  the  collected  MetalMapper 
data  cannot  be  classified.  These  anomalies  must  be  placed  in  the  dig  category,  which  reduces  the 
effectiveness  of  the  classification  process. 

3.4.1  Metric 

The  number  of  anomalies  for  which  reliable  parameters  cannot  be  estimated  was  the  metric  for 
this  objective. 


3  It  should  be  noted  that  when  “probability”  is  referenced  in  this  document,  it  is  not  a  statistical  probability,  but 
rather  a  percent.  The  term  is  used  to  remain  consistent  with  the  work  plans  and  scoring  reports  created  under  the 
SERCP/ESTCP  program. 
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3.4.2  Data  Requirements 

The  requirement  was  that  those  targets  for  which  parameters  were  not  reliably  estimated  were  to 
be  identified  as  such  on  the  prioritized  dig  list  submitted  following  analysis  of  the  MetalMapper 
data. 

3.4.3  Success  Criteria 

The  objective  was  to  estimate  reliable  parameters  for  >  98%  of  the  targets  on  the  prioritized  dig 
list. 


3.5  OBJECTIVE:  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

This  objective  involves  the  accuracy  of  the  target  parameters  that  were  estimated  in  the  first 
phase  of  the  analysis.  Successful  classification  was  only  possible  if  the  input  features  were 
internally  consistent.  The  clearest  way  to  satisfy  this  condition  was  to  estimate  the  various  target 
parameters  accurately. 

3.5.1  Metric 

Accuracy  of  estimation  of  target  parameters  was  the  metric  for  this  objective. 

3.5.2  Data  Requirements 

The  requirement  was  that  target  parameters  were  to  be  provided  as  part  of  the  final  results 
submittal.  IDA  analysts  compared  these  estimated  parameters  to  those  measured  during  the 
intrusive  investigation  and  detennined  via  subsequent  in-air  measurements. 

3.5.3  Success  Criteria 

The  objective  was  to  estimate  X,  Y  locations  within  15  cm  (Ict)  and  the  estimated  depths  within 
10  cm  (la),  of  the  true  location. 
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4.0  SITE  DESCRIPTION 


The  MetalMapper  survey  was  conducted  in  two  areas  at  the  fonner  Camp  Beale.  One  area  was 
utilized  for  only  MetalMapper  surveying  and  the  other  was  utilized  as  a  combined  survey  area. 
The  combined  area  was  surveyed  using  MetalMapper  as  well  as  the  other  advanced  classification 
systems  evaluated  as  part  of  the  ESTCP  Live  Site  Demonstration  at  the  former  Camp  Beale. 

The  MetalMapper-only  survey  area  consisted  of  22  contiguous  30-meter  by  30-meter  grids.  A 
total  of  1,213  geophysical  targets  were  surveyed  with  the  MetalMapper  in  this  area.  The  survey 
area  was  predominately  grass  covered  with  moderately  steep  slopes  along  the  northeastern  edge 
of  the  survey  area.  The  site  was  relatively  open  and  contained  a  minimal  number  of  rock 
outcrops.  The  combined  survey  area  consisted  of  4  contiguous  30-meter  by  30-meter  grids.  A 
total  of  257  geophysical  targets  were  surveyed  with  the  MetalMapper  in  this  area.  It  was 
relatively  open,  grass  covered,  moderately  to  steeply-sloped  and  contained  a  minimal  number  of 
rock  outcrops.  The  survey  areas  at  the  former  Camp  Beale  were  selected  by  the  ESTCP  Program 
Office.  Survey  benchmarks  for  RTK-GPS  base  station  locations  had  also  been  established  by  the 
ESTCP  Program  Office  during  earlier  phases  of  the  Live  Site  Demonstration. 

Suspected  munitions  within  the  survey  areas  at  the  former  Camp  Beale  include,  but  may  not  be 
limited  to,  the  following:  37  mm  projectiles,  60  mm  mortars,  81  mm  mortars,  and  105  mm 
projectiles. 

Additional  infonnation  on  the  site  history,  description,  survey  areas,  and  site  selection  criteria 
can  be  found  in  the  ESTCP  Live  Site  Demonstrations  Plan  (ESTCP,  2011). 
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5.0  SURVEY  EXECUTION 


5.1  SITE  PREPARATION 

The  survey  areas  were  prepared  by  ESTCP  in  advance  of  the  MetalMapper  survey,  so  no  special 
site  preparation  activities  (e.g.  vegetation  removal,  grass  cutting,  etc.)  were  necessary  at  the  start 
of  the  MetalMapper  survey.  Vinyl-stem  flags  marked  the  locations  of  the  geophysical  targets  in 
the  combined  survey  grids  because  several  of  the  other  advanced  classification  systems  evaluated 
at  former  Camp  Beale  do  not  have  real-time  GPS  positioning  capabilities  like  the  MetalMapper. 
No  flags  were  located  within  the  MetalMapper-only  grids. 

An  instrument  verification  strip  (IVS)  was  constructed  by  ESTCP  during  an  earlier  phase  of  the 
Live  Site  Demonstration  at  former  Camp  Beale.  Items  buried  within  the  IVS  included  the 
following:  steel  sphere,  105-HEAT,  37  mm  projectile,  60  mm  mortar  and  a  small  industry 
standard  object  (ISO).  Information  on  burial  depth,  orientation  and  geographic  coordinates  of 
the  IVS  items  can  be  found  in  the  ESTCP  Live  Site  Demonstrations  Plan  (ESTCP,  2011). 

5.2  ON-SITE  TRAINING 

On-site  training  on  the  MetalMapper  operation  was  provided  to  CH2M  HILL  by  Geometries 
personnel  on  June  20,  2011.  On-site  data  analysis  training  was  provided  by  Snyder  Geoscience, 
Inc.  between  June  21  and  23,  2011. 

5.3  CALIBRATION  AND  SYSTEM  FUNCTIONAL  CHECKS 

Daily  calibration  and  system  functional  checks  were  perfonned  as  part  of  routine  quality 
assurance/quality  control  (QA/QC)  procedures.  These  checks  were  performed  at  or  near  the  IVS 
location. 

Twice  daily  measurements  of  the  IVS  were  made  with  the  MetalMapper.  These  data  were 
provided  each  day  to  the  onsite  CH2M  HILL  analyst  and  were  used  to  evaluate  the  stability  of 
the  MetalMapper  system  over  the  duration  of  the  survey  as  well  as  the  response  over  specific, 
known  items.  In  addition,  measurements  along  the  IVS  were  recorded  in  order  to  evaluate  the 
accuracy  of  the  RTK-GPS  positional  data  throughout  the  duration  of  the  survey. 

Twice  daily  measurements  over  an  established  background  location  near  the  IVS  were  also 
recorded  as  part  of  the  daily  system  functional  checks.  This  location  was  a  shallow  test  pit  that 
had  been  excavated  under  the  direction  of  ESTCP  and  detennined  to  be  relatively  free  of 
subsurface  metal.  These  data  were  used  to  evaluate  the  system  response  at  a  consistent  location 
relatively  free  of  subsurface  metal  throughout  the  duration  of  the  survey. 

Calibration  data  collected  at  fonner  Camp  Beale  included  a  series  of  measurements  over  inert 
items  placed  in  the  test  pit.  The  items,  burial  depth,  and  orientation  are  provided  in  Table  5-1. 
These  measurements  were  used  primarily  as  training  data  and  to  develop  a  target  classification 
library  to  be  used  during  data  analysis. 
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Table  5-1:  Calibration  Measurements  in  Test  Pit 


Item 

Depth 

(cm) 

Orientation 

Empty  pit  (static  -  5  min) 

N/A 

N/A 

Steel  sphere 

25 

N/A 

37  mm  projectile 

10,  20 

Horizontal  (along  track  and  across  track), 
45°  down,  90°  down 

60  mm  mortar 

20,30 

Horizontal  (along  track  and  across  track), 
45°  down,  90°  down 

8 1  mm  mortar 

25,40 

Horizontal  (along  track  and  across  track), 
45°  down,  90°  down 

105  mm  projectile 

30,  45 

Horizontal  (along  track  and  across  track), 
45°  down,  90°  down 

5.4  DATA  COLLECTION  PROCEDURES 

The  acquisition  parameters  utilized  for  the  survey  at  former  Camp  Beale  are  presented  in  Table 
5-2.  The  parameters  were  established  based  on  experience  during  previous  MetalMapper 
demonstration  surveys.  The  CH2M  HILL  data  collection  team  consisted  of  two  on-site  staff. 
Field  work  was  performed  in  accordance  with  the  Accident  Prevention  Plan  (APP)  and  Site 
Specific  Health  and  Safety  Plan  (SSHP)  (CH2M  HILL,  201 1). 


Table  5-2:  Acquisition  Parameters 


Mode 

Tx 

Mode 

Hold-Off 
Time  (ns) 

Block 

Period 

(s) 

Rep  Fctr 

Dec 

Fctr 

(%) 

Stk 

Const 

Base 

Freq 

(Hz) 

Decay 
Time  (us) 

No. 

Gates 

Sample 
Period  (s) 

Sample 
Rate  (S/s) 

Static 

ZYX 

50 

0.9 

27 

10 

10 

30 

8333 

50 

9 

N/A 

Definitions: 

Tx  Mode:  Transmitter  mode.  ZYX  =  All  three  transmitters  active 

Hold-Off  Time:  Relates  to  calculating  gates;  the  50ps  hold-off  means  that  the  2nd  gate  starts  no  earlier  than  50ps.  Gates  are  calculated  starting 
with  the  delay  time  of  the  last  sample.  The  last  gate  (50th)  has  a  width  of  8333.33  x  0.1  ps  calculated  to  the  nearest  sample  integer  sample 
interval  of  4ps. 

Block  Period:  Data  are  acquired  during  a  specified  time  period  (in  seconds)  called  a  block  period. 

Rep  Fctr:  The  number  of  transmitter  periods  per  block  period.  The  repetition  factor  will  always  be  a  power  of  3  (e.g.,  1,3,9,27,. . .) 

Dec  Fctr:  Decimation  Factor;  a  decimal  fraction  or  percentage  used  to  calculate  the  gate  widths  as  a  function  of  delay  time  to  the  end  of  the  time 
gate  under  constmction.  Width  =  Dec  Fctr  *  Gate  End  Time.  The  Width  is  rounded  to  an  integer  number  of  ADC  sample  intervals  (4  us). 

Base  Freq:  Base  Frequency;  this  is  the  fundamental  frequency  of  the  transmitter  waveform.  Base  Frequency=  Rep  Factor/Block  Time  (e.g. 
27/0.9  =  30Hz) 

No.  Gates:  Number  of  gates;  gate  widths  and  center  times  are  calculated  starting  with  the  sample  with  the  longest  delay  time  using  the 
decimation  factor  to  figure  out  the  gate  width.  The  process  proceeds  from  late  to  early  gates  and  stops  when  the  start  of  the  next  gate  is  less  than 
the  delay  time.  There  is  always  one  more  gate  (the  1  st  gate)  than  the  aforementioned  procedure  allows.  The  first  gate  is  used  for  a  composite  gate 
(i.e.,  an  average  of  several  gates).  The  parameters  of  that  gate  are  contained  in  the  AcqParams  block  of  the  TEM  data  file.  It  is  not  used  for  static 
data  interpretation. 

Sample  Period:  The  Block  Time  *  Stk  Constant. 
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5.4.1  Production  Survey 

MetalMapper  data  acquisition  was  perfonned  over  1,470  targets.  The  geophysical  target 
locations  were  provided  by  ESTCP  prior  to  the  start  of  the  MetalMapper  survey.  Seed  item 
locations  were  included  in  the  target  list,  their  locations  known  only  to  ESTCP.  The  target  list 
was  fonnatted  for  input  into  the  MetalMapper  data  acquisition  software  EM3DAcquire, 
developed  by  G&G  Sciences,  Inc. 

The  acquisition  software  displays  a  map  of  the  target  locations  with  their  assigned  unique  IDs, 
where  after  a  target  is  recorded,  the  software  displays  an  “X”  on  top  of  the  target  location,  thus 
enabling  the  operator  to  know  which  points  have  been  collected.  Each  day,  the  complete  list  of 
points  recorded  to-date  was  loaded  on  to  the  map  to  avoid  unnecessary  recollection  of  targets. 
The  software  also  provided  a  real-time  display  of  the  GPS  quality  parameters  and  the  IMU 
readings,  where  potential  problems  with  these  data  streams  could  be  identified  before  taking  a 
measurement. 

With  the  acquisition  parameters  in  Table  5-2,  the  length  of  time  to  record  a  single  target  location 
was  approximately  30  seconds. 

5.4.2  Background  Acquisition 

Periodic  background  readings  were  taken  in  static  mode  with  the  MetalMapper  each  day  during 
data  collection.  Background  readings  were  taken  approximately  every  60  to  90  minutes  and 
were  intended  to  evaluate  spatial  changes  in  background  conditions  across  the  survey  area  as 
well  as  to  evaluate  whether  background  levels  changed  over  time  as  a  result  of  decreased  battery 
voltage  or  sensor  heating  during  the  course  of  each  day  in  the  field. 

The  background  locations  were  identified  in  real-time  using  the  MetalMapper  EM3DAcquire 
acquisition  software.  The  software  facilitates  the  identification  of  suitable  background  locations 
by  providing  a  real-time  graphical  display  of  sensor  response  amplitude  and  position  relative  to 
nearby  metallic  targets.  Suitable  background  locations  were  those  where  amplitudes  appeared  to 
be  minimal  and  where  there  did  not  appear  to  be  nearby  metallic  sources. 

5.4.3  Data  Quality  Checks 

In  addition  to  the  aforementioned  calibration  and  system  functional  checks,  qualitative  checks  of 
the  data  were  performed  using  the  plot  functions  of  the  EM3DAcquire  acquisition  software.  The 
transmitter  output  current  and  2 1  receiver  measurements  were  periodically  generated  for  the  raw 
survey  data  in  order  to  identify  potential  system  malfunctions. 

A  final  check  on  the  quality  of  static  data  was  completed  during  preliminary  analysis  with  the 
program  MM/RMP.  This  final  check  was  generally  perfonned  within  24  hours  of  data 
collection,  and  if  necessary,  targets  identified  for  re-collection  were  provided  to  the  field  team. 
For  the  survey  at  the  former  Camp  Beale,  a  total  of  1 10  targets  were  identified  for  re-collection. 

Re-collection  was  typically  a  result  of  the  lateral  offset  of  the  preliminary  modeled  target 
location  falling  outside  the  established  positional  tolerance  from  the  center  of  the  MetalMapper 
array.  High  fidelity  estimates  of  a  target’s  principal  polarizability  curves  depend  on  adequate 
illumination  of  the  target  along  each  of  its  principal  axes.  A  conservative  guideline  is  that  targets 
with  horizontal  offsets  of  40  cm  or  more  from  the  center  of  the  Z  transmitter  were  not  adequately 
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illuminated  and  thus  their  symmetry  properties  (i.e.,  one  major  polarizability  curve  over  two 
nearly  identical  minor  polarizability  curves)  may  not  be  apparent.  For  the  survey  at  the  former 
Camp  Beale,  the  positional  tolerance  was  therefore  established  at  40  cm. 

Additional  reasons  for  re-collection  also  included,  to  a  lesser  extent,  poor  GPS  data  quality  of  the 
recorded  target  location  or  weak  signal  response  due  to  site  conditions  (e.g.  rock  outcrop  at  target 
location)  or  the  potential  impact  from  multiple  targets  located  within  the  array  platform  footprint. 

5.4.4  Data  Handling 

Data  were  recorded  in  binary  format  to  the  hard  disk  of  the  MetalMapper  DAQ.  Raw  data  files 
were  downloaded  daily  and  transmitted  to  the  project  analyst  along  with  relevant  field  notes. 
Raw  and  pre-processed  data  were  provided  to  ESTCP  upon  completion  of  the  survey.  Raw,  pre- 
processed,  and  final  processed  data  are  kept  on  file  at  CH2M  HILL. 

The  file  prefix  nomenclature  utilized  at  the  former  Camp  Beale  survey  areas  was  as  follows: 
T=target  location;  X=daily  IVS  measurement;  B=periodic  background  reading;  PIT=test  pit 
measurement.  This  nomenclature  was  used  because  the  acquisition  software  stores  file  names  in 
sequential  order.  The  unique  target  ID  provided  by  ESTCP  is  embedded  within  the  data  file, 
although  this  ID  is  not  reflected  in  the  actual  file  name.  At  the  request  of  ESTCP,  the  unique  IDs 
for  targets  requiring  re-collection  were  increased  by  10,000  (e.g.  recollected  target  ID  1202 
would  be  saved  as  target  1 1202). 

5.5  VALIDATION 

At  the  conclusion  of  data  collection  activities,  all  geophysical  targets  were  excavated  by  ESTCP. 
Each  item  was  identified,  photographed,  its  depth  measured,  its  position  recorded  using  RTK- 
GPS,  and  the  item  was  removed,  if  possible.  All  non-hazardous  items  were  saved  by  ESTCP  for 
later  in-air  measurements.  This  ground  truth  information  was  provided  by  ESTCP  and  was  used 
to  further  validate  the  performance  objectives  listed  in  Section  3.0. 
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6.0  DATA  ANALYSIS 


Analysis  and  interpretation  of  the  MetalMapper  data  collected  at  the  former  Camp  Beale 
involved  the  following  steps:  pre-processing,  parameter  estimation,  analysis  of  training  data,  and 
classification. 

6.1  PRE-PROCESSING 

Collected  data  were  preprocessed  using  the  MM/RMP  software  package.  MM/RMP  was  used  to 
convert  the  recorded  geographic  coordinates  to  Universal  Transverse  Mercator  (UTM) 
coordinates  (Zone  10  North),  correct  the  survey  location  point  using  attitude  data  from  the  IMU 
and  remove  the  background  field  from  the  receiver  transients.  Pre-processed  data  for  each  target 
were  exported  as  a  .CSV  file.  CH2M  HILL  imported  preprocessed  data  into  Geosoft’s  Oasis 
Montaj  processing  environment  for  further  analysis. 

6.2  PARAMETER  ESTIMATION 

Parameter  estimation  was  perfonned  using  the  UX-Analyze  module  in  Oasis  Montaj.  Target 
data  were  inverted  using  both  single-source  and  multi-source  dipole  response  models  to  estimate 
target  parameters.  The  principle  parameters  of  interest  for  use  in  classification  of  the  targets 
were  the  three  polarizabilities  (Pi,  P2,  and  P3)  estimated  for  each  target  by  UX-Analyze.  In 
addition  to  estimates  for  the  three  Ps  for  each  target,  an  estimated  location  and  depth  was  also 
returned  by  UX-Analyze  for  each  target  during  inversion.  Targets  modeled  outside  the  positional 
tolerance  of  40  cm  from  the  center  of  the  MetalMapper  array  were  identified  for  re-collection 
(see  Section  5.4.3). 

While  not  used  extensively  in  a  classification  survey  where  all  geophysical  targets  are  intrusively 
investigated,  the  location  and  depth  are  significant  when  considering  their  effects  on  the  success 
of  an  intrusive  investigation  based  on  MetalMapper  results.  The  same  is  true  for  inclination, 
depth,  and  rotation  of  the  source  object,  which  are  also  estimated  by  UX-Analyze.  For  the 
analysis  of  the  former  Camp  Beale  data,  these  parameters  were  evaluated,  although  they  were  not 
a  metric  for  the  survey  performance  objectives. 

6.3  ANALYSIS  OF  TRAINING  DATA 

Training  data  included  data  that  was  collected  during  previous  ESTCP  MetalMapper  surveys  as 
well  as  the  test  pit  data  collected  by  CH2M  HILL  described  in  Section  5.3.  These  data  were  used 
to  create  a  library  of  polarizabilities  for  various  standard  munitions  types.  CH2M  HILL  elected 
to  include  additional  munitions  types  not  known  to  have  been  used  at  Camp  Beale  in  the  library 
of  polarizabilities.  This  decision  was  based  primarily  on  the  ESTCP  Munitions  Response  Live 
Site  Demonstrations  Plan,  Section  4.3,  (ESTCP,  2011),  which  states  that  the  suspected  munitions 
in  this  demonstration  area  include,  but  are  not  limited  to  37  mm  projectiles,  60mm  mortars, 
81mm  mortars  and  105mm  projectiles,  but  that  due  to  the  complex  history  of  the  site  and 
overlapping  network  of  historical  ranges,  it  is  also  likely  that  other  munitions  types  beyond  those 


13 


listed  above  may  be  encountered.  As  a  result,  CH2M  HILL  elected  to  include  the  following 
additional  items  in  the  library: 

•  1 5  5  mm  proj  ectiles 

•  2.36in  rockets  (intact,  tail  booms,  and  rocket  motors) 

•  2.75in  rockets 

•  20mm  projectiles 

•  25mm  projectiles 

•  3 in  Stokes  mortar  projectiles 

•  4.2in  mortar  projectiles 

•  40mm  projectiles 

•  57mm  projectiles 

•  75  mm  projectiles 

•  76mm  projectiles 

•  BDU-28s 

•  BDU-33s 

•  M42  bomblet 

•  M48  fuzes 

•  MK  118  Rockeyes 

•  MK23  practice  bomb 

•  MK  II  grenades 

•  Small  ISOs  (included  because  were  used  in  IVS) 

The  library  of  polarizabilities  was  compared  to  the  recovered  MetalMapper  target  polarizabilities 
from  Camp  Beale  for  classification  purposes.  At  the  request  of  CH2M  HILL,  additional  training 
data  were  provided  by  ESTCP  for  49  geophysical  targets  for  use  during  the  classification 
process. 

6.4  CLASSIFICATION 

The  classifier  used  to  create  the  ranked  dig  list  for  the  former  Camp  Beale  targets  was  based 
primarily  on  the  confidence  metric  generated  by  UX -Analyze  during  a  comparison  of  the  P 
values  estimated  for  each  surveyed  target  and  the  P  values  in  the  munitions  library  developed  for 
the  project.  The  confidence  metric  indicates  the  fit  ratio  between  a  target  and  the  items  in  the 
library,  with  higher  metrics  indicating  a  better  fit  between  the  target  and  the  corresponding  item 
in  the  library.  Targets  with  a  confidence  metric  of  0.80  or  better  are  typically  considered  a  good 
fit  to  the  library  item. 

Comparison  of  target  data  to  the  library  was  performed  using  the  Advanced  Target  Classification 
function  of  UX-Analyze,  which  allows  the  user  to  weight  the  importance  of  each  of  the  three 
polarizability  curves  for  the  purposes  of  the  comparison.  The  first  comparison  perfonned 
weighted  all  three  polarizabilities  equally.  Following  this  initial  comparison,  all  relatively  high 
confidence  metrics  generated  were  considered  indicative  of  TOI.  Preliminary  classification  was 
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based  on  a  fit  ratio  of  0.80  as  the  cut-off  between  targets  likely  to  be  TOI  and  targets  identified  as 
either  non-TOI  or  those  for  which  an  initial  decision  could  not  be  made.  This  initial  classification 
was  then  refined  as  described  below. 

Targets  with  high  fit  ratios  (greater  than  0.80)  to  library  items  using  all  three  polarizabilities  were 
classified  as  Category  1  (dig  on  first  pass)  targets.  Their  position  on  the  ranked  dig  list  was 
detennined  in  reverse  order  of  confidence  metric  (i.e.  higher  confidence  metrics  are  of  higher 
priority  on  the  list).  Targets  with  the  same  confidence  metrics  were  ranked  according  to  the 
response  amplitude  from  the  EM61-MK2  data,  with  higher  amplitude  anomalies  given  priority 
over  lower  amplitude  anomalies. 

Lower  confidence  metrics  could  be  indicative  of  either  non-TOI  sources  or  of  poor  quality 
MetalMapper  data  due  to  low  signal-to-noise  or  the  inability  to  position  the  MetalMapper  sensor 
array  directly  over  the  target  location  (e.g.  due  to  obstacles  such  as  rock  outcrops).  Two 
additional  categories  were  identified  using  the  first  comparison  of  MetalMapper  data  to  library 
data:  1)  targets  for  which  a  decision  cannot  be  made,  and  2)  targets  likely  to  be  non-TOI.  The 
threshold  separating  these  two  classes  was  initially  set  at  a  confidence  metric  of  0.60.  Targets 
were  repositioned  on  the  dig  list  based  on  further  individual  analysis,  as  described  below. 

Following  initial  categorization,  the  target  to  library  polarization  curve  matches  for  all  targets  not 
assigned  to  Category  1  were  examined  by  the  analyst.  The  goal  of  this  examination  was  to 
identify  targets  for  which  poor  MetalMapper  data  quality,  despite  the  effort  to  recollect 
questionable  data,  was  the  potential  cause  for  a  lower  confidence  metric  rather  than  simply  a 
poor  match  due  to  a  non-TOI  source.  Targets  showing  poor  polarizability  fits  were  noted  for 
further  analysis  if  any  of  the  three  polarization  curves  appeared  to  be  of  sufficient  quality  that 
it/they  could  be  used  for  comparison  to  the  library  data.  If  none  of  the  three  curves  appeared 
useful,  the  target  was  classified  as  Category  0  (cannot  analyze)  and  was  added  to  the  top  of  the 
ranked  dig  list. 

Targets  for  which  one  or  two  polarization  curves  appeared  useful  were  compared  to  the  library  a 
second  time,  with  the  (3(s)  exhibiting  poor  data  quality  removed  from  the  comparison  (i.e.  it/they 
were  given  0  weights).  The  targets  involved  in  the  reanalysis  were  re-categorized  and  re-ranked 
on  the  dig  list  based  on  the  revised  confidence  metrics.  The  confidence  metric  thresholds  used  to 
differentiate  likely  TOI  from  likely  clutter  were  different  for  one  or  two  curve  comparisons  than 
those  used  for  three  curve  comparisons.  These  thresholds  were  evaluated  following  one-  or  two- 
curve  analysis.  Following  completion  of  the  various  comparisons  of  targets  data  to  the  library 
data,  a  dig/no  dig  threshold  was  identified  on  the  dig  list.  The  threshold  was  within  Category  2, 
those  targets  for  which  a  TOFnon-TOI  decision  was  not  able  to  be  made. 

The  initial  ranked  dig  list  was  submitted  for  comparison  to  the  results  of  the  intrusive 
investigation.  In  accordance  with  the  ESTCP  Live  Site  Demonstrations  Plan  (ESTCP,  2011),  the 
results  of  the  intrusive  investigation  for  those  targets  classified  as  “dig”  on  the  ranked  dig  list 
were  supplied  following  submittal  of  the  list  to  allow  the  demonstrator  to  analyze  the  results  and 
revise  the  dig/no  dig  threshold  and/or  ranking  of  each  target.  CH2M  HILL  did  not  revise  the 
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initial  dig  list  following  this  analysis  but  elected  to  use  the  same  rankings  for  the  final  ranked  dig 
list  submitted  to  ESTCP  for  scoring. 

6.5  DELIVERABLES 

The  following  deliverables  resulted  from  the  data  collection  at  Camp  Beale: 

1)  Final  Prioritized  Dig  List:  Detailed  in  Section  6.4. 

2)  Cued  Data:  Text  readable  cued  data  files  were  provided  in  2  fonns:  a)  raw  data,  and  b) 
pre-processed  data.  The  following  operations  were  performed  on  the  pre-processed  cued 
data  files: 

i)  Coordinate  Conversion:  GPS  lat/lon  was  converted  to  UTM  Zone  ION,  NAD83 
coordinates. 

ii)  Coordinate  Corrections:  Using  cart  attitude  angles  (heading,  pitch,  and  roll)  the 
UTM  coordinates  was  corrected  to  MM  platform  reference  point. 

iii)  Background  Removal:  An  appropriate  background  was  removed  from  all  the 
receiver  transients.  Thus,  the  transients  written  to  the  file  were  estimates  of  the 
secondary  fields  after  background  had  been  removed. 

3)  Cross-Reference  List:  A  text  readable  table  that  associates  MM  filenames  with  each 
Target  ID. 
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7.0  PERFORMANCE  ASSESSMENT 


The  performance  objectives  for  this  classification  survey  and  the  corresponding  results  are 
summarized  in  Table  7-1.  Details  on  the  results  for  each  objective  are  subsequently  discussed  in 
the  following  sections. 

Table  7-1 :  MetalMapper  Survey  Results  for  Former  Camp  Beale 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Objective 

Met? 

Maximize  correct 
classification  of 
targets  of  interest 
(TOI) 

Number  of  TOI 
retained. 

Prioritized  anomaly 
lists 

Approach  correctly 
classifies  all  TOI 

No 

Maximize  correct 
classification  of 
non-TOI 

Number  of  false 
alarms  eliminated. 

Prioritized  anomaly 
lists 

Reduction  of  false 
alarms  by  >  50% 
while  retaining  all 

TOI 

No 

Specification  of  no¬ 
dig  threshold 

Probability  of  correct 
classification  and 
number  of  false 
alarms  at 
demonstrator 
operating  point. 

Demonstrator  - 
specified  threshold 

Reduction  of  false 
alarms  by  >  50% 
while  retaining  all 

TOI 

No 

Minimize  number 
of  anomalies  that 
cannot  be  analyzed 

Number  of  anomalies 
that  must  be 
classified  as  “Unable 
to  Analyze.” 

Demonstrator  target 
parameters 

Reliable  target 
parameters  can  be 
estimated  for  >  98% 
of  anomalies 
detection  list. 

No 

Correct  estimation 
of  target 
parameters 

Accuracy  of 
estimated  target 
parameters. 

•  Demonstrator 
target  parameters 

•  Results  of 
intrusive 
investigation 

X,  Y  <15  cm  (la) 

Z  <10  cm  ( 1  a) 

No 

7.1  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOI 

One  of  the  two  main  objectives  of  this  demonstration  was  to  correctly  classify  all  seeded  items 
and  any  MEC  items  remaining  at  the  site  as  TOI  (i.e.,  identify  true  positives,  with  no  false 
negatives). 

The  cued  MetalMapper  survey  and  subsequent  data  analysis  and  classification  failed  to  identify 
all  TOI,  and  thus  the  performance  objective  was  not  met.  Seven  TOI  were  not  identified  for 
excavation:  three  small  ISOs,  two  37mm  projectiles,  and  two  105mm  projectiles.  The  inverted 
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data  of  these  missed  TOI  did  not  appear  to  match  well  with  any  ordnance  items,  and  they  were 
therefore  classified  as  non-TOI. 

The  missed  TOI  are  discussed  in  the  following  subsections. 

7.1.1  Target  BE-1500  -  105mm  projectile 

Although  UX-Analyze  returned  a  0.8016  confidence  metric  match  between  target  BE-1500  and  a 
105mm  projectile,  the  polarizability  curves  did  not  visually  match  those  of  a  typical  105mm 
projectile  (Figure  7-1,  with  the  recovered  [3s  in  bold  colors,  and  the  Ps  of  the  closest  library 
matches  in  faded  colors).  The  recovered  primary  P  decayed  more  quickly  than  the  corresponding 
P  of  previously  seen  105mm  projectiles,  and  the  recovered  minor  Ps  roughly  paralleled  the 
primary  beta  throughout  the  decay  curve  rather  than  falling  off  at  a  quicker  rate  as  seen  in  typical 
105mm  projectiles.  Target  BE-1500  was  therefore  classified  as  a  non-TOI. 
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Figure  7-1:  Target  BE-1500  Classification  Results 
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Closest  Library  Matches  for  Target  BE-1500 


Typical  105mm  Projectile  ps 


7.1.2  Target  BE-1723  -  37mm  projectile 

As  with  Target  BE-1500,  the  recovered  polarizability  curves  of  target  BE-1723  did  not  visually 
match  those  of  a  typical  37mm  projectile  (Figure  7-2,  with  the  recovered  ps  in  bold  colors,  and 
the  ps  of  the  closest  library  matches  in  faded  colors).  The  amplitude  of  the  recovered  primary  P 
appeared  lower  than  that  typically  seen  in  37mm  projectiles,  and  the  recovered  minor  Ps  roughly 
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paralleled  the  primary  beta  throughout  the  decay  curve  rather  than  falling  off  at  a  quicker  rate  as 
seen  in  typical  37mm  projectiles.  Target  BE- 1723  was  therefore  classified  as  a  non-TOI. 


Figure  7-2:  Target  BE-1723  Classification  Results 
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7.1.3  Target  BE-1892  -  105mm  projectile 


The  recovered  polarizabilities  of  target  BE-1892  do  not  appear  to  match  those  of  a  105mm 
projectile  or  any  other  common  ordnance  item  (Figure  7-3,  with  the  recovered  [3s  in  bold  colors, 
and  the  [3s  of  the  closest  library  matches  in  faded  colors).  The  shape  of  the  recovered  primary  P 
matches  fairly  well  with  the  expected  primary  P  shape  of  a  105mm  projectile,  but  the  amplitude 
is  much  lower  than  that  seen  in  typical  105mm  projectiles.  The  secondary  and  tertiary  Ps  bare  no 
resemblance  to  those  of  a  105mm  projectile.  The  classification  library  used  in  this  analysis 
contained  numerous  ordnance  items  in  addition  to  those  items  expected  at  the  site  (37mm 
projectiles,  60mm  mortars,  81mm  mortars  and  105mm  projectiles)  and  therefore  returned  a 
correspondingly  large  number  of  fit  matches.  In  order  to  filter  the  returned  data  to  a  manageable 
quantity,  the  analyst  reviewed  only  the  top  six  matches  for  each  target.  The  fit  between  target 
BE-1892  and  a  105mm  projectile  from  the  library  was  not  good  enough  to  be  displayed  as  one  of 
the  top  six  matches,  and  the  vague  similarity  that  did  exist  was  therefore  not  apparent. 
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Figure  7-3:  Target  BE-1892  Classification  Results 
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7.1.4  Target  BE-1951  -  37mm  projectile 

Target  BE-1951,  a  37mm  projectile,  could  have  been  identified  as  a  target  without  data  of 
sufficient  quality  to  be  properly  analyzed.  The  two  recovered  polarizabilities  (Figure  7-4,  with 
the  recovered  (3s  in  bold  colors,  and  the  (3s  of  the  closest  library  matches  in  faded  colors)  are 
likely  not  strong  enough  for  confident  classification  but  due  to  their  dissimilarity  to  any  typical 
ordnance  item  were  incorrectly  classified  as  non-TOI. 
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Figure  7-4:  Target  BE-1951  Classification  Results 
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7.1.5  Target  BE-1960  -  small  ISO 

As  with  target  BE- 1892,  the  failure  to  identify  target  BE-1960  as  a  small  ISO  is  due,  in  part,  to  a 
classification  library  that  includes  ordnance  items  not  found  at  Camp  Beale.  The  two  best 
matches  of  the  recovered  polarizabilities  (Figure  7-5,  with  the  recovered  ps  in  bold  colors,  and 
the  |3s  of  the  closest  library  matches  in  faded  colors)  were  with  20mm  projectiles,  which  were  not 
found  at  the  site.  The  remainder  of  the  six  viewed  matches  were  with  37mm  projectiles,  and  no 
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matches  with  small  ISOs  were  returned.  The  shape  of  the  primary  [3  does  mimic  the  expected 
shape  of  a  small  ISO’s  primary  (3,  but  the  amplitude  is  considerably  lower  than  that  of  a  typical 
small  ISO.  The  low  amplitude  of  the  [3s  coupled  with  the  lack  of  symmetry  displayed  in  the 
minor  [3s  caused  the  match  to  a  small  ISO  to  fall  out  of  the  top  six  displayed  library  matches,  and 
thus,  the  vague  similarity  was  not  apparent. 

Figure  7-5:  Target  BE-1960  Classification  Results 
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7.1.6  Target  BE-1965  -  small  ISO 


The  recovered  primary  polarizability  of  target  BE-1965,  and  to  a  lesser  extent  the  recovered 
secondary  P  (Figure  7-6,  with  the  recovered  Ps  in  bold  colors,  and  the  Ps  of  the  closest  library 
matches  in  faded  colors)  match  the  expected  p  shapes  of  a  small  ISO  very  well,  but  the 
amplitudes  are  considerably  smaller,  which  typically  indicates  a  smaller  item.  The  low 
amplitudes  of  the  Ps  resulted  in  this  target  being  classified  as  a  non-TOI. 


Figure  7-6:  Target  BE-1965  Classification  Results 
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7.1.7  Target  BE-2091  -  small  ISO 


The  recovered  polarizabilities  of  target  BE-2091  (Figure  7-7,  with  the  recovered  Ps  in  bold 
colors,  and  the  Ps  of  the  closest  library  matches  in  faded  colors),  a  small  ISO,  do  not  provide  a 
good  match  with  an  ISO  or  any  typical  ordnance  item.  Neither  the  amplitude  nor  the  shape  of 
the  Ps,  match  well,  and  the  target  was  therefore  classified  as  a  non-TOI. 


Figure  7-7:  Target  BE-2091  Classification  Results 
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7.2  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-TOI 


This  was  the  second  of  the  two  primary  measures  of  the  effectiveness  of  the  classification 
survey.  In  addition  to  correctly  classifying  TOI,  the  effectiveness  of  the  MetalMapper  in 
discriminating  munitions  is  a  function  of  the  degree  to  which  responses  that  do  not  correspond  to 
TOI  can  be  eliminated  from  consideration  during  the  intrusive  investigation  (i.e.,  the  ability  to 
identify  true  negatives,  with  minimal  false  positives). 

This  perfonnance  objective  was  not  met  since  not  all  TOI  were  retained  above  the  dig  threshold; 
however,  81%  of  the  non-TOI  items  were  correctly  labeled.  The  objective  was  to  be  considered 
met  if  more  than  50%  of  the  non-TOI  items  were  correctly  labeled  as  non-TOI  while  retaining  all 
of  the  TOI  above  the  dig  threshold.  Of  the  1,244  non-TOI  items,  1,013  were  correctly  labeled. 

Of  the  1,470  analyzed  targets,  29  were  “shared”  targets,  where  one  item  caused  multiple  EM61- 
MK2  anomalies  selected  for  MetalMapper  analysis.  As  a  result,  only  1,441  of  the  1,470  cued 
MetalMapper  targets  had  unique  sources.  Of  the  1,441  unique  targets,  428  were  identified  to  be 
excavated,  including  49  training  items  and  44  targets  that  could  not  be  reliably  analyzed.  This 
resulted  in  77  recovered  munitions  items  and  27  recovered  ISO  items.  The  remaining  213  items 
within  the  “dig”  category  were  non-TOI. 

7.3  OBJECTIVE:  SPECIFICATION  OF  NO-DIG  THRESHOLD 

Through  retrospective  analysis,  it  is  possible  to  evaluate  the  true  capabilities  of  a  classification 
procedure  based  solely  on  the  ranked  anomaly  list.  In  a  scenario  where  all  geophysical  targets 
may  not  be  intrusively  investigated,  the  success  of  the  procedure  will  depend  on  the  ability  of  the 
analyst  to  accurately  specify  the  dig/no-dig  threshold.  The  objective  was  considered  met  if  the 
threshold  could  be  chosen  to  achieve  the  previous  two  objectives.  This  amounted  to  controlling 
the  Type  II  error  (false  negative  rate)  for  TOI  at  zero  while  also  controlling  the  Type  I  error 
(false  positives)  for  Clutter  at  less  than  50%. 

All  TOI  were  not  retained  above  the  dig  threshold.  Therefore,  the  analyst  was  unable  to  set  a 
threshold  that  met  this  objective. 
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Based  on  the  Receiver  Operating  Curve  (ROC)  curve  created  from  CH2M  HILL’s  prioritized  dig 
list  (Figure  7-8),  it  would  not  have  been  possible  to  choose  a  threshold  that  controlled  the  Type  II 
error  (false  negatives  for  TOI)  at  the  specified  tolerance  (zero)  without  increasing  the  Type  I 
errors  (false  positives  for  Clutter)  beyond  the  specified  tolerance  (>50%). 


Difficult  UXOs: 

#BE-1951  (37mm  stats  seed) 
#BE-2091  (ISO  stats  seed) 
#BE-1965  (ISO  stats  seed) 
#BE-1892  (105mm  stats  seed) 
#BE-1723  (37mm  stats  seed) 
#BE-1960  (ISO  stats  seed) 
#BE-1500  (105mm  stats  seed) 


Figure  7-8:  ROC  Curve  for  Camp  Beale  MetalMapper  Survey. 


In  Figure  7-8,  the  black  points  at  either  end  of  the  curve  represent  hypothetical,  extreme  choices 
of  the  dig  threshold.  Specifically,  the  black  point  at  the  upper,  right  end  of  the  curve  (just  visible 
under  the  gold  dot)  represents  the  situation  where  the  dig  threshold  was  placed  at  the  very  bottom 
of  the  ranked  anomaly  list,  such  that  all  anomalies  are  declared  “Dig”.  Conversely,  the  black 
point  near  the  lower,  left  end  of  the  curve  represents  the  extreme  situation  in  which  the  dig 
threshold  is  placed  near  the  top  of  the  ranked  anomaly  list,  in  between  the  training  and 
calibration  data  (shown  as  the  solid  black  portions  of  the  curve),  such  that  only  the  training  data 
are  declared  “Dig”  while  all  calibration  data  are  declared  “Do  Not  Dig”. 


In  addition  to  the  black  dots,  the  curve  contains  two  large  dots  representing  two  specific  dig 
thresholds.  The  blue  dot  represents  the  stop-dig  point  chosen  by  the  analyst.  The  orange  dot 
represents  the  retrospectively  chosen  “best  case  scenario”  dig  threshold,  which  would  have 
minimized  the  false  positives  for  Clutter  while  ensuring  that  false  negatives  for  TOI  were  zero. 
Listed  to  the  right  of  the  curve  are  the  TOIs  that  were  incorrectly  classified  (the  false  negatives) 
at  the  analyst’s  dig  threshold  (blue  dot). 
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7.4  OBJECTIVE:  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE 
ANALYZED 

Anomalies  for  which  reliable  parameters  cannot  be  estimated  using  the  collected  MetalMapper 
data  cannot  be  classified.  These  anomalies  must  be  placed  in  the  dig  category,  which  reduces  the 
effectiveness  of  the  classification  process. 

This  objective  was  to  be  considered  met  if  reliable  parameters  were  estimated  for  >  98%  of  the 
targets  on  the  prioritized  dig  list. 

Of  the  1,441  unique  targets,  44  could  not  be  reliably  analyzed.  This  amounts  to  an  estimation 
rate  of  96.9%,  which  fails  to  meet  the  >98%  criterion. 

7.5  OBJECTIVE:  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

This  objective  involves  the  accuracy  of  the  target  parameters  that  were  estimated  in  the  initial 
stages  of  the  data  analysis.  Successful  classification  was  only  possible  if  the  modeled  features 
were  an  accurate  representation  of  the  item. 

The  objective  was  to  be  considered  met  if  the  estimated  X,  Y  locations  were  within  15  cm  (la), 
and  the  estimated  depths  were  within  10  cm  (la)  of  the  true  location.  Generally,  the  X,  Y 
location  was  estimated  within  39  cm  of  the  true  location.  Ground  truth  information  on  depths 
was  not  made  available;  nevertheless,  this  objective  is  not  met  on  the  X,  Y  criterion  alone. 

7.6  PERFORMANCE  ASSESSMENT  SUMMARY 

Although  the  performance  objectives  of  the  demonstration  were  not  met,  the  demonstration 
provided  valuable  “hands-on”  experience  and  worked  to  further  the  process  of  transferring  the 
MetalMapper  sensor  and  target  classification  methods  from  the  researchers  and  developers  to  the 
end-users.  The  required  precision  of  the  survey  and  the  volume  of  data  generated  for  each 
surveyed  target  culminate  in  a  steep  learning  curve  for  both  the  successful  collection  of  the  data 
and  its  processing  and  analysis.  Nevertheless,  this  project  demonstrated  that  industry  personnel 
with  little  to  no  previous  experience  with  the  system  can  quickly  learn  to  collect  data,  analyze 
results,  and  classify  anomalies. 

Specific  lessons  learned  in  this  demonstration  include  the  following: 

•  It  is  important  to  tailor  the  library  of  polarizability  curves  to  fit  the  project  site  as  much  as 
possible.  Although  unexpected  ordnance  types  are  always  possible  at  an  unknown  site,  if 
certain  munitions  can  be  reasonably  excluded  from  the  library  in  advance,  a  more  refined 
library  with  fewer  individual  items  may  reduce  the  number  of  multiple  matches  to  the 
same  type  of  ordnance.  This  would  allow  the  analyst  to  more  easily  view  the  fit  of  the 
target  to  multiple  ordnance  types  and  to  eliminate  the  easy  non- fits. 

•  Feedback  from  site-specific  excavation  results  is  critical  as  further  training  for  the 
classification  process. 
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•  Matches  to  library  items  must  be  considered  for  cases  where  only  one  or  two  of  the 
recovered  polarizabilities  fit  those  of  a  library  item. 

•  The  shape  of  the  polarizability  curve(s)  must  be  considered  even  when  their  amplitudes 
do  not  fit  a  library  item  (and  vice  versa). 

•  TOI  may  not  always  exhibit  perfect  axial  symmetry  (i.e.  the  secondary  and  tertiary 
polarizability  curves  may  not  be  perfectly  identical). 

•  Additional  demonstration  projects  and  hands-on  experience  will  enable  users  to  improve 
the  ability  to  collect  data,  analyze  results,  and  classify  anomalies  and  thereby  work  to 
effectively  transfer  the  technology. 
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8.0  COST  ASSESSMENT 


ESTCP  projects  are  required  to  develop  and  validate,  to  the  extent  possible,  the  expected 
operational  costs  of  the  technology.  The  intent  of  this  section  is  to  identify  the  infonnation  that 
was  tracked  or  the  data  that  was  obtained  during  the  demonstration  that  will  aid  in  establishing 
realistic  costs  for  implementing  the  technology  and  comparing  it  to  potential  alternative 
technologies.  The  tracked  costs  are  provided  in  Table  8-1  and  discussion  on  the  cost  elements 
provided  in  the  following  subsections.  CH2M  HILL  was  not  involved  in  preparation  of  the  site, 
so  costs  are  only  provided  for  data  collection  and  data  processing.  Note  that  the  MetalMapper 
system  and  the  tractor  used  to  tow  the  system  were  provided  by  ESTCP  (through  a  contract  with 
Geometries)  at  no  cost  to  CH2M  HILL  for  this  demonstration,  thus  only  daily  rates  provided  by 
vendors  or  estimated  are  provided  for  those  cost  elements.  The  cost  per  anomaly  for  data 
collection  also  does  not  include  those  costs. 

Table  8-1 :  Costs  for  MetalMapper 


!  Cost  Element 

Tracked  Data 

Cost/Quantity 

Data  Collection  Costs 

Pre/Post  Survey 
Activities 

Total  (does  not  include  estimated  costs  for  components 
provided  by  others  demarcated  by  “*”) 

$6000 

•  MetalMapper  mobilization  (Geometries  fee)* 

$1000 

•  MetalMapper  daily  rate* 

$500 

•  Transportation  (ground)* 

$2000 

•  Tractor  mobilization/demobilization* 

$500 

•  Personnel  mobilized 

3 

•  Personnel  mobilization/ demobilization  (days) 

2 

•  Personnel  training  (days) 

1 

•  Setup  and  test  pit  surveys  (days) 

1 

Unit  cost  per  anomaly  investigated  (does  not  include 
estimated  costs  for  components  provided  by  others 
demarcated  by  “*”) 

$16.21 

Survey  Costs 

•  MetalMapper  daily  rate* 

$500 

•  Tractor  daily  rate* 

$200 

•  Survey  personnel  used 

3 
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•  Personnel  hours  per  day 

10 

•  Daily  setup  and  calibration 

1 

•  Average  number  of  anomalies  surveyed  per  day  (re¬ 
surveys  are  not  counted  as  additional  anomalies) 

235 

•  Estimated  additional  cost  per  anomaly  for 

MetalMapper  and  tractor 

$1.50 

•  Daily  equipment  break-down  and  storage  (hours) 

0.5 

Processing  Costs 

Preprocessing  (in  the 
field) 

Unit  cost  per  anomaly  investigated 

$4.57 

Data  processing 

(office) 

Unit  cost  per  anomaly  investigated 

$6.72 

*  The  MetalMapper  system  and  the  tractor  used  to  tow  the  system  were  provided  by  ESTCP  at  no  cost  to  CH2M 
HILL  for  this  demonstration,  thus  these  costs  are  based  on  estimates  or  vendor  quoted  rates. 


8.1  DATA  COLLECTION  COSTS 

For  the  purposes  of  tracking,  the  data  collection  costs  were  broken  out  into  two  sub-categories: 
(1)  Pre/Post  Survey  and  Activities  and  (2)  Survey  Costs. 

8.1.1  Pre/Post  Survey  and  Activities 

CH2M  HILL’s  total  cost  for  the  Pre/Post  Survey  and  Activities  sub-category  includes  personnel 
time  for  mobilization  and  demobilization,  training  with  the  system  (provided  by  Geometries 
personnel),  setup  and  calibration  (test  pit  surveys),  and  all  associated  travel  costs.  Although  the 
MetalMapper  used  on  this  project  was  not  paid  for  from  CH2M  HILL’s  project  budget, 
Geometries  was  contacted  for  current  rental  rates  when  the  cost  analysis  was  performed  and  the 
daily  cost  is  provided.  The  cost  to  ship  the  MetalMapper  from  Geometries  home  office  to  the 
project  site  was  estimated  for  this  project;  however,  costs  to  ship  can  vary  considerably  based  on 
distance  from  the  shipping  location  and  the  shipping  method.  Shipping  of  the  MetalMapper  by 
air  is  a  significant  cost  because  some  elements  of  the  system  (e.g.  the  sled)  cannot  be  broken 
down  into  smaller  components. 

The  tractor  used  to  transport  the  MetalMapper  system  around  the  site  was  also  provided  by 
Geometries.  An  estimated  daily  cost  has  been  provided  in  Table  8-1;  however,  this  cost  will  also 
vary  depending  on  survey  locations  and  available  equipment. 

8.1.2  Survey  Costs 

CH2M  HILL’s  survey  costs  include  all  personnel  hours  and  associated  travel  costs  for  daily 
setup,  system  testing,  anomaly  survey  activities,  and  break-down/storage.  Again,  the 
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MetalMapper  and  tractor  used  to  transport  the  MetalMapper  system  around  the  site  was  provided 
by  Geometries.  The  personnel  costs  were  divided  by  the  number  of  anomalies  on  the  target  list  to 
arrive  at  the  per  anomaly  cost  for  surveying.  The  total  number  of  cued  points  collected  was  not 
used,  as  re-shots  are  an  expected  necessity  on  most  projects.  The  extra  time  required  for 
reinvestigation  of  points  with  poor  data  quality  was,  therefore,  factored  into  the  per  anomaly 
cost. 


8.2  PROCESSING  COSTS 

For  the  purposes  of  tracking,  the  processing  collection  costs  were  broken  out  into  two  sub¬ 
categories:  (1)  Preprocessing  and  (2)  Processing. 

8.2.1  Preprocessing 

Preprocessing  costs  included  the  personnel  hours  (and  associated  travel  costs  since  the 
processing  geophysicist  was  in  the  field)  needed  to  transfonn  a  raw  (binary)  survey  file  into  a 
more  usable  .CSV  file,  including  any  background  data  removal  and  necessary  QC  checks.  The 
cost  for  hardware  and  software  for  performing  the  preprocessing  was  not  included  in  the  costs  as 
these  are  typically  overhead  costs.  The  personnel  costs  were  divided  by  the  number  of  anomalies 
on  the  target  list  to  arrive  at  the  per  anomaly  cost  for  preprocessing.  The  total  number  of  cued 
points  collected  was  not  used,  as  re-shots  are  an  expected  necessity  on  most  projects.  The  extra 
time  required  for  reprocessing  of  points  with  poor  data  quality  was,  therefore,  factored  into  the 
per  anomaly  cost. 

8.2.2  Processing 

Processing  costs  included  the  personnel  hours  to  perform  all  aspects  of  the  data  analysis,  from 
import  of  the  individual  target  .CSV  files  through  compilation  of  the  final  ranked  dig  list.  Data 
processing  was  performed  out  of  the  field  and  the  cost  for  hardware  and  software  for  performing 
the  preprocessing  was  not  included  in  the  costs  as  these  are  typically  overhead  costs.  The 
personnel  costs  were  divided  by  the  number  of  anomalies  on  the  target  list  to  arrive  at  the  per 
anomaly  cost  for  processing. 
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